System and method for enabling the creation of personalized movie presentations and personalized movie collections

ABSTRACT

A system and method for providing on demand access to information related to a movie while the movie is being presented to a user, where the movie was generated from the movie related information. The system operates by presenting the movie to the user, and then receiving from the user a query pertaining to the movie. The system determines a frame of the movie that was being presented to the user when the user issued the query (the system may extract this information from the query, or may extract this information from the movie itself). The system identifies, as specified by the query, portions of the movie related information relating to the frame, and retrieves those portions of the movie related information. These retrieved portions of said movie related information are presented to the user. The system includes a manager to enable users to create personalized versions of movies, and personalized collections of items. The system includes a system and method for providing to users on demand access to merchandise information related to a movie, and for providing on demand connectivity to merchants, while the movie is being presented to the users.

This application is a division of application Ser. No. 08/407,305, filedMar. 20, 1995, (status: pending).

CROSS-REFERENCE TO OTHER APPLICATIONS

The following applications of common assignee contain some commondisclosure, and are believed to have an effective filing date identicalwith that of the present application.

"System and Method for Linking and Presenting Movies With TheirUnderlying Source Information," application Ser. No. 08/407,306,incorporated herein by reference.

"System and Method for Providing Merchant Information and EstablishingLinks to Merchants While Presenting a Movie," application Ser. No.08/407,307, incorporated herein by reference.

DESCRIPTION

1. Technical Field

The present invention relates generally to data linking andpresentation, and more particularly to linking and presenting movieswith their underlying source information.

2. Related Art

The Information Age that currently exists has produced incredibleamounts of data that are currently unavailable to the public. In manycases, the producers of such information would be more than willing toprovide the information to the public for some reasonable fee, and thepublic would be willing to pay this fee. Typically, however, theseinformation producers (also called information providers) are not ableto take advantage of this significant business opportunity due to theproblems inherent in providing the information to the public.

One problem relates to the medium in which the information is stored.Today, information is stored on paper, on film, on video, on compactdisks, on magnetic tapes, on computer floppy disks, etc. Informationrelated to a particular topic may be stored on any combination of thesestorage mediums. In order to provide information on that topic to anuser, it would be necessary for an information provider to identify andlocate all items containing information related to the topic, and thentransport these items to the user. The value of these items to the useris questionable, since the user probably does not possess the equipmentneeded to access the information stored in all of the items.Accordingly, it is difficult to provide information to users becausesuch information is stored in many different storage mediums.

A more significant problem results from the sheer volume of informationrelating to any particular topic. Such information is of little value toan user unless the user is able to efficiently and expeditiously makeuse of the information. For example, suppose that the user is reviewinga document written by an author and decides that it would be interestingto study other works the author has had a part in creating. Theinformation provided to the user from the information provider will beof little value unless the user is able to efficiently, effectively, andexpeditiously traverse the information and locate the data on the authorof interest.

These above-described commercial opportunities associated with thedissemination of information, and the problems that prevent suchcommercial opportunities from being realized, shall now be describedmore concretely by considering a real-life example: the motion pictureindustry.

Viewing a movie via a theatrical presentation today involves travellingto the theater, waiting in line to buy a ticket, choosing a seat, andwaiting for the movie to begin. The viewer's schedule must match that ofthe theater. Once the movie has started, the viewer must watchpromotional previews of other films before watching the feature film.During the presentation, the viewer has no control over the sound volumeor presentation flow. When the movie is over, the viewer travels backhome. The picture quality and sound are excellent, and the viewer enjoysthe best viewing experience possible today. However, these benefits areobtained only at the expense of convenience and viewing control. Thus,commercial opportunities exist in electronically transferring movies toviewer's homes.

The technology to electronically transfer movies to viewer's homesexists today. Viewing a movie over a conventional network or premiumchannel broadcast is simpler and more convenient, but the viewer hasfewer movie choices and enjoys a more limited quality experience. Like atheatrical viewing, the viewer's schedule must match that of thebroadcast and the viewer does not control the presentation flow. Thereis no travelling involved, and the cost is generally less than that of atheater experience. The video and audio quality are dependent on theviewer's television and sound system. Note that, with this viewingoption (and also with the theatrical presentation option discussedabove), the viewer is typically not provided with any supplementalinformation about the movie, such as information about the making of themovie, interviews with the director and cast members, information on theactors, information about special effects, etc.

Alternatively, a person can watch a movie by renting a video tape orlaser disk. Watching a movie in this manner involves two trips to therental shop, a search of their available titles, a wait in line, and arental fee. The viewing experience is improved over that of a broadcastpresentation, as the viewer has some presentation controls (pause, fastforward, rewind, etc.). A laser disk may also offer supplementalinformation about the film. Parallel audio tracks may be includedcontaining, for example, director commentary. Separate chapters of videomay also be provided. However, there is no ability to play video contentfrom different chapters simultaneously. The cost is generally higherthan that of a network broadcast. Cost comparisons to that of a premiumchannel are dependent on usage patterns (flat fee versus fee perrental).

A mid-90s `near video on demand` viewing experience represents somethingbetween that of broadcast movies and a VCR/laser disk. A viewer selectsa movie from an expanded broadcast rotation (generally a much smallercollection than that available at a VCR rental outlet). The viewer mustwait a few minutes until the next rotation of the movie broadcast. Apurchase request via remote control or an 800 number is required toenable viewing. Once the presentation has begun, the viewer has alimited range of controls over the presentation flow. If the movie ispaused, the viewer must wait until the next instance of the movie atthat point is available in the rotation. This latency is dependent onthe number of channels available and the collection size configured bythe viewer's cable carrier, and is typically about five minutes. Notravel is required, and the per use fee is typically added to theviewer's monthly cable bill. Supplemental information about the movie istypically not available.

A mid-90s `true video on demand` viewing experience is effectively likethat of a VCR rental, without two trips to the rental outlet. A digitalmovie server and ITV infrastructure are used to enable the viewer toselect a movie from a large collection of all digitized films availableon the server. The available selection list is not restricted by abroadcast rotation, and each viewer has full and immediate control overtheir movie presentation. Like `near video on demand,` a purchaserequest via remote control or an 800 number is required to enableviewing. The per use fee is typically added to the viewer's monthlybill, and supplemental information is typically not available.

Thus, the current options for viewing a movie are limited because theyare inconvenient (the viewer must travel to the theater or to the videostore), and/or provide the viewer with little or no presentation flow.Also, these current viewing options are limited because they provide theviewer with little or no supplemental movie information. Thus, themotion picture industry is not taking advantage of the potentialcommercial opportunities associated with providing supplemental movieinformation to viewers. Even in those cases where supplementalinformation is provided, the viewer has little control over how suchsupplemental information is presented. Accordingly, the supplementalinformation is available only in a limited way to videophiles who arewilling to pay extra for "special" or "director's cut" editions of thetitles.

DISCLOSURE OF INVENTION

The present invention is directed to a system and method for providingon demand access to information related to a movie while the movie isbeing presented to a user, where the movie was generated from the movierelated information. The invention operates by presenting the movie tothe user, and then receiving from the user a query pertaining to themovie. The invention determines a frame of the movie that was beingpresented to the user when the user issued the query (the invention mayextract this information from the query, or may extract this informationfrom the movie itself). The invention identifies, as specified by thequery, portions of the movie related information relating to the frame,and retrieves those portions of the movie related information. Theseretrieved portions of the movie related information are presented to theuser.

The present invention is also directed to a system and method forenabling a user to create, modify, and utilize a personalized version ofa movie. The system includes a foundation information database whichstores movie related information used to generate the movie. Thisembodiment of the invention operates by enabling the user to modify acopy of an edit decision list associated with the movie. The editdecision list includes an entry for each scene, clip, and take in themovie. The modified edit decision list defines the personalized versionof the movie. The personalized version of the movie is presented to theuser in response to a request to do so from the user. Such presentationoperates by retrieving an entry from the modified edit decision list,retrieving from the foundation information database one or moredigitized frames specified by the retrieved entry, and presenting thedigitized frames to the user.

In a similar manner, the present invention is directed to a system andmethod for enabling a user to create, modify, and utilize a personalizedcollection of items. This embodiment of the invention operates byenabling the user to modify a personalized collection table, where thepersonalized collection table defines the personalized collection andhas an entry for each item in the personalized collection. Thepersonalized collection of items is presented to the user in response toa request to do so from the user. Such presentation operates byretrieving an entry from the personalized collection table, retrievingfrom the foundation information database foundation informationspecified by the retrieved entry, and presenting the retrievedfoundation information to the user. The items contained in thepersonalized collection include any combination of shots, takes, scenes,clips, audio segments, etc.

Further, the present invention is directed to a system and method forproviding on demand access to merchandise information related to amovie, and for providing on demand connectivity to merchants, while themovie is being presented to a user. This embodiment of the inventionoperates by presenting the movie to the user, receiving from the user amerchandise related query pertaining to the movie, and determining ascene of the movie that was being presented to the user when the userissued the query. Portions of movie related information relating tomerchandise appearing in the scene are identified and retrieved. Theseportions of movie related information are then presented to the user.The invention enables the user to interact with any merchant whoprovided any merchandise appearing in the scene in response to a requestfrom the user to do so.

Further features and advantages of the present invention, as well as thestructure and operation of various embodiments of the present invention,are described in detail below with reference to the accompanyingdrawings. In the drawings, like reference numbers indicate identical orfunctionally similar elements.

The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescription of (a) preferred embodiment(s) of the invention, asillustrated in the accompanying drawing(s).

BRIEF DESCRIPTION OF FIGURES

The present invention will be described with reference to theaccompanying drawings, wherein:

FIG. 1 is a block diagram of a preferred embodiment of the presentinvention;

FIG. 2 is a block diagram of a computer system representing a preferredimplementation of many components of the present invention;

FIGS. 3 and 4 are data flow diagrams depicting the manner in whichfoundation and index information is generated according to a preferredembodiment of the present invention;

FIGS. 5 and 6 illustrate example index information contents;

FIG. 7 illustrates the pedigree of a frame;

FIG. 8 illustrates an example source table;

FIGS. 9A, 9B, 11, 15, 17A, 18, 19, and 21 are flowcharts depicting theoperation of the present invention;

FIGS. 10A-10C, 12A, 12B, 13A, 13B, 14A, 14B, 17B, 20, 22A, and 22B areexample index tables; and

FIG. 16 is a block diagram of a manager according to a preferredembodiment of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

The present invention is directed to a system and method for organizinginformation, and for providing services to enable users to efficiently,effectively, and expeditiously access such information. In other words,the present invention provides services to enable users to accessinformation in an intelligent, user-driven manner.

For illustrative purposes, the present invention is sometimes describedherein in the context of information produced by the motion pictureindustry. That is, the present invention is described as a system andmethod for organizing movie and supplemental movie information, and forpresenting such movie and supplemental movie information to users. Itshould be understood, however, that the present invention is not limitedto this embodiment (call the "movie environment" for referencepurposes). Instead, the present invention is applicable to anyenvironment where information is produced, and where a commercialopportunity exists with the organization and presentation of suchinformation. For example, the present invention is applicable tocomputer programming environment where information about the person whowrote a program, the company that sold the program, the place theprogram was written and tested, etc., could be linked to a runningprogram.

The movie embodiment of the present invention is preferably realizedthrough a combination of a `true video on demand` infrastructure,database technology, and other specialized digital servers. By tightlycoupling databases to specialized digital servers for video, audio,still image, text, etc., a novel level of control of movie presentationis achieved. Additional information about a movie is stored and managed,to permit a viewer to more effectively explore and understand movies.The combination of greater control and additional information permitsthe viewer to experience a new level of richness and entertainment frommovies.

Accordingly to the movie embodiment, text, still image, and audioinformation is directly related to specific movie scenes or frames. Thislevel of association, coupled with a database, permits a greater levelof direct access to movie information than previously possible.Searching and correlation of supplemental information (also called sideinformation) permits the examination of different side informationrelationships, and permits the following sample actions not existing inconventional viewing technologies:

1. List the names of the actors, other credited roles, or scenetechnical detail for the current scene (e.g., director, producer,musical score, shot location, blue screen, matte, etc.).

2. List other movies, TV shows, or other scenes in the current moviewith one or more properties equal to those found above.

3. List all scenes including a specific product tie-in (i.e., productbrand name used in movie).

4. Save any of these lists for viewing selection.

5. View scene while listening to voice overs of director or actor withtheir comments about the scene.

6. View stills with pan and zoom controls, to examine fine detail.

7. View movie at desired censor level.

8. View script or staging information via picture in picture.

9. View outtakes for the current scene.

10. View signing (i.e., sign language) via picture in picture.

11. Search other stills for like colors, shapes, or textures.

12. Search the script for a word or phase.

13. Search other scripts for like words or phrases.

14. Layout a movie presentation box, with areas for displaying relatedinformation of interest.

15. View a movie through a presentation box, to concurrently view themovie and related information of interest.

In addition to these queries predominately based on script and editinglinkages, queries can be made on non-text factors. Metrics extractedfrom the video, audio, or stills are also managed by the database, andare used to form non-text searches against these data. This scheme canbe used to search for non-text aspects such as cut boundaries and movieshaving similar soundtracks.

The movie embodiment of the present invention also permits theexamination of movie detail beyond that viewable through a TV or HDTV.Feature films are today captured principally on high grade film,affording fidelity far beyond that viewable on TV or HDTV. The presentinvention permits a viewer to pan and zoom, to examine finer detail in amovie or image still.

By using the movie embodiment of the present invention, film makers areafforded a new level of flexibility in optional viewing and associationsfrom within their films. Alternate plot lines, cuts, or censor levelsare much more easily constructed, because the film components aremanipulated independently. Composition is directed through the database,easing the construction of alternate presentations and linkages.

STRUCTURE OF THE PRESENT INVENTION

The present invention shall now be discussed more generally withreference to FIG. 1, which illustrates a block diagram of a dataprocessing environment 102 of the present invention. This environment102 includes a plurality of user devices 106 representing, for example,set top boxes (STB) in combination with television monitors, orcomputers (such as personal computers). The user devices 106 eachinclude a mechanism (such as a keyboard either integrated into the userdevices 106 or contained in a remote control unit) for receiving controland information messages from human operators.

Each user device 106 also includes a well known transmit and receivecomponent to transmit and receive control and information signals fromother devices via a communication medium 124. Preferably, thecommunication medium 124 represents a data communication network, suchas a local area network or a wide area network, in which case theenvironment 102 represents a distributed computing environment. However,the present invention is not limited to this connection scheme. Forexample, some of the devices shown in FIG. 1 may be positioned locallyto each other, and/or some of the devices in FIG. 1 may be implementedusing a single computer. In sum, any mechanism or scheme for connectingthe devices shown in FIG. 1 is within the scope and spirit of thepresent invention.

The environment 102 also includes a foundation information interfacecomponent 108 that controls access to foundation information in afoundation information database 112. (As shown by line 110, thefoundation information interface component 108 is preferably directlyconnected to the foundation information database 112.) The term"foundation information" is used herein to refer to information that hasbeen produced by one or more parties (each party being a person ororganization). According to the present invention, the foundationinformation represents digital information only. The present inventionprovides foundation information on demand to users via the user devices106. With respect to the movie embodiment, the foundation informationrepresents, for example, a movie and supplemental information pertainingto the movie (i.e., additional information generated during thepre-production, production, and post-production stages of the movie).The foundation information is described in greater detail below.

Preferably, the foundation interface component 108 represents arelational database management system (RDBMS). The foundationinformation database 112 preferably represents a file server. Thefoundation information database 112 may be distributed for performancereasons (i.e., to position foundation information closer to the userdevices 106 so that communication performance is enhanced), in whichcase there would be multiple foundation information interface components108 and multiple foundation information databases 112 (for simplicitypurposes, only one foundation information interface component 108 andone foundation information database 112 are described).

An index interface component 118 controls access to index information inan index information database 122. (As shown by line 120, the indexinterface component 118 is preferably directly connected to the indexinformation database 122.) This index information represents indices ofthe foundation information in the foundation information database 112.Accordingly, the organization of the foundation information isestablished by the index information. The present invention uses theindex information to locate and reference particular foundationinformation. The index information is discussed further below.

Preferably, the index interface component 118 represents a relationaldatabase management system (RDBMS). The index information database 122preferably represents a file server which has special searchingcapabilities or another relational database management system (RDBMS).The index information 122 may be distributed for performance reasons, inwhich case there would be multiple index interface components 118 andmultiple index information databases 122 (for simplicity purposes, onlyone index interface component 118 and one index information database 122are described).

The environment 102 further includes a presentation and controlcomponent 104. The functionality of the presentation and controlcomponent 104 may be distributed, in which case there would be multiplepresentation and control components 104. For simplicity purposes, onlyone presentation and control component 104 is discussed. Thepresentation and control component 104 coordinates the actions of theuser devices 106, the foundation information interface component 108,and the index interface component 118.

In particular, the presentation and control component 104 receivesinformation requests from the user devices 106. The presentation andcontrol component 104 processes these information requests by accessingthe index information in the index information database 122 (via theindex interface component 118) to identify and locate the particularfoundation information indicated by the user requests. The presentationand control component 104 then retrieves this foundation informationfrom the foundation information database 112 (via the foundationinformation interface component 108), and presents this foundationinformation to the users via the user devices 106. In an alternativeembodiment, any or all of the functionality of the presentation andcontrol component 104 is embedded in the index interface component 118and/or the foundation information interface component 108, such that theuser devices 106 (in some circumstances) communicate directly with theindex interface component 118 and/or the foundation informationinterface component 108.

In an embodiment of the present invention, the user devices 106 are eachused as the application driver in a client/server model, and makesserver requests of the databases 112, 122 and digital servers throughthe index interface component 118 and the foundation informationinterface component 108 through an integrated interface. Thefunctionality of this integrated interface is represented by thepresentation and control component 104, although such functionality canalternatively be located or distributed among other modules. Throughthese interfaces, the digital video, audio, and still image objectsstored on the digital server(s) appear to the application to be managedas data in the relational databases 108, 118.

The index information database 122 and the foundation informationdatabase 112 are preferably implemented as digital servers for at leasttwo reasons:

1. To permit the use of specialized hardware and software tocost-effectively deliver the digital video, audio, and image stills.

2. To permit the distribution of these servers, thereby reducing thedistance to the viewer and the associated latency and transmissioncosts.

As noted above, each user device 106 can be a full function PC, or amore limited function settop box. Depending on the hardware and softwarecapacity of any particular user device 106, the client applicationexecuting thereon may be distributed over the user device 106 and anintermediate station emulating a full function end station. Independentof this distribution of function, the user device/intermediate stationinteracts with both the relational databases 112,122 and the digitalmovie servers 108, 118 through the integrated interface (i.e., thepresentation and control component 104).

The environment 102 also includes a cross merchant gateway 116 and amerchant transaction processor 114, which are described in U.S. PatentApplication entitled "System and Method for Providing MerchantInformation and Establishing Links to Merchants While Presenting aMovie", application Ser. No. 08/407,307, referenced above.

The user devices 106., foundation information interface component 108,index interface component 118, presentation and control component 104,foundation information database 112, index information database 122,cross merchant gateway 116, and merchant transaction processor 114 areeach preferably implemented using a computer system 202 as shown in FIG.2. (Alternatively, each computer system 202 is used to implement asubset of these devices/components.) The computer system 202 includes aprocessor 204 (or multiple processors 204) communicating with othercomponents via a communication bus 206.

Connected to the bus 206 is a primary memory 208 (such as random accessmemory, RAM) which stores control logic 210 (i.e., software) and data212 (such as a portion of the foundation information and indexinformation, for example). During run-time, the control logic 210enables the processor 204 to perform the functions described herein.Thus, the control logic 210 represents a controller of the processor204. In an alternate embodiment, the functionality of the presentinvention described herein is achieved using a hardware implementedstate machine.

The computer system 202 also includes secondary memory 214, which mayinclude a storage device 216, such as a removable disk drive (i.e., afloppy drive). A removable storage medium 218, such as a computer disk(i.e., a floppy disk) can be inserted in the storage device 216. Thecontrol logic 210 is stored in direct computer readable form (i.e.,magnetically or optically) on a storage medium (i.e., magnetic storagemedium) in the removable storage medium 218.

Also connected to the bus 206 is a network interface 222, which providesconnectivity to the communication medium 124 shown in FIG. 1.

Source, Foundation, and Index Information

The present invention also includes a manager 1602 which is shown inblock diagram form in FIG. 16. The manager 1602 includes one or morecapture and digitizer components 304 (for simplicity purposes, only oneis described), one or more index generation components 308 (forsimplicity purposes, only one is described), a personalized presentationmanager 1604, and a personalized collection manager 1606. The manager1602 and its components 304, 308, 1604, 1606 are preferably implementedusing one or more computer-based devices operating in accordance withinstructions from computer software (as shown in FIG. 2). Alternatively,the functionality of at least portions of the manager 1602 (as describedherein) is achieved using predominately hardware, such as a hardwareimplemented state machine.

The personalized presentation manager 1604 and the personalizedcollection manager 1606 are discussed below. The capture and digitizercomponent 304 and the index generation component 308 shall now bedescribed.

The capture and digitizer component 304 and the index generationcomponent 308 are used to generate the foundation information that isultimately stored in the foundation information database 112, and theindex information that is ultimately stored in the index informationdatabase 122. The detailed operation of the capture and digitizerComponent 304 and the index generation component 308 shall now bedescribed with reference to a data flow diagram shown in FIG. 3.

The capture and digitizer component 304 receives source information 302.The source information 302 represents information that has been producedover time by one or more parties (each party being a person ororganization), and that would be of value to some end users (i.e., thosepersons working at the user devices 106). This information may be storedin any number of storage mediums, such as paper, film, video, compactdisk, magnetic tape, computer storage disks, etc.

An example of the contents of the source information 302 in the movieembodiment of the present invention is illustrated in FIG. 4. FIG. 4also illustrates the process by which the source information 302 isgenerated in the movie embodiment.

As will be appreciated by persons skilled in the relevant art(s), amotion picture is generally produced in three phases: a pre-productionphase 402, a production phase 406, and a post-production phase 410.Source information 302 is generated during each of these phases.Specifically, pre-production information 404 is generated during thepre-production phase 402, production information 408 is generated duringthe production phase 406, and post-production information 412 isgenerated during the post-production phase 410.

The pre-production information 404 includes, for example, diagrams ofmodels, storyboards, contracts with actors and other persons andorganizations associated with the movie, information on locations,versions of scripts, etc. The production information 408 includes, forexample, takes, continuity sheets, department work orders, etc.

The post-production information 412 includes, for example, one or moreedit decision lists 414. An edit decision list 414 specifies the scenes,takes, and/or clips that are in the theatrical version of the movie.There may be multiple versions of a movie (for example, the director'sversion, the unrated version, a signed version, etc.). Generally, anedit decision list 414 exists for each version.

The post-production information 412 also includes the video 416 andaudio 418 associated with each version of the movie, and clockinformation 420 that is embedded in the video 416 and audio 418. Eachframe in a movie has a time code. These time codes represent the clockinformation 420. The clock information 420 and time codes are discussedin greater detail below.

The edit decision list 414, video information 416, audio information418, and clock information 420 represent primary information 492,whereas the pre-production information 404 and the productioninformation 408 represent secondary information 490 (also called movierelated information). Primary information 492 is what is normallypresented and available to the public. Its creation is the purpose forperforming a process such as that shown in FIG. 4. Secondary information490 is generated for the purpose of generating the primary information492, and/or is generated during the creation of the primary information492, and/or is related to the primary information 492. In other words,the primary information 492 is generated from the secondary information490. The secondary information 490 is not generally presented to oravailable to the public. However, often the public would be veryinterested in this secondary information 490, and would be willing topay a fee to have access to it. The present invention is directed toproviding the public with access to this secondary information 492 in anintelligent, user-friendly manner. Although the concepts of primaryinformation 492 and secondary information 490 have been explained withreference to the movie embodiment, it should be understood that primaryinformation 492 and secondary information 490 are generated in othercontexts, and the present invention is equally applicable to these othercontexts.

The pre-production information 404, production information 408, andpost-production information 412 of the source information 302 can bestored in any number of storage mediums. For example, diagrams ofmodels, storyboards, contracts with actors and other persons, locationinformation, and department work orders are typically stored on paper.Scripts are also stored on paper, but may additionally be stored inelectronic form on computer storage disks (i.e., floppy disks). An editdecision list 414 is stored on paper, and may additionally be stored inthe electronic form used by automated offline editing tools. Takes arestored on film. Video 416 is stored on film and/or VCR tapes. Audio 418is stored on tape, compact disks, computer storage disks, etc.

Referring again to FIG. 3, the capture and digitizer component 304converts the source information 302 to foundation information 306. Thefoundation information 306 has the same information content as thesource information 302, but is in digital form. Thus, the foundationinformation 306 is a digital, electronic representation of the sourceinformation 302.

Specifically, the capture and digitizer component 304 captures orotherwise extracts the source information 302 from the mediums on whichthey are stored, digitizes the captured/extracted source information302, and stores such digitized information as the foundation information306. Preferably, the operation of the capture and digitizer component304 is automated to the greatest extent possible, but may require somehuman intervention and control depending on available technology.

The capture and digitizer component 304 includes, for example, scanners,optical character recognition (OCR) devices and processes, datacompressors and decompressors, data extract and capture devices andprocesses, data converters, etc. The structure and operation of thesedevices and processes are well known to persons skilled in the relevantart(s) and, thus, shall not be described further.

In some cases, at least some portions of the source information 302 arealready in digital, electronic form. Such portions of the sourceinformation 302 are directly stored as foundation information 306, asindicated by data flow line 312.

Eventually, it is possible that most or even all of the sourceinformation 302 will be available in digital, electronic form (once thisoccurs, the role of the capture and digitizer component 304 will bereduced or even eliminated). This is true for at least two reasons.First, the advance in computer technology is making it more and moreattractive for parties to perform their work using a computer. As aresult, that work is stored in digital, electronic form. Second,information generators who intend to utilize the present invention willhave an incentive to store their work in digital, electronic form (theincentive being the ability to eliminate the capture and digitizercomponent 304).

The index generator component 308 analyzes the foundation information306 and produces index information 310. The operation of the indexgenerator component 308 is automated to the greatest extent possible,but may include user intervention and control (such as keying in indexinformation). The index information 310 represents indices of thefoundation information 306. Accordingly, the organization of thefoundation information 306 is established by the index information 310.For example, suppose the foundation information 306 includes works by agroup of authors, and articles written about those works and authors. Inthis example, the index information 310 may have a first table listingby author works available in the foundation information 306, a secondtable listing articles available in the foundation information 306 abouteach author, and a third table listing articles available in thefoundation information 306 about each work.

With regard to the movie embodiment, and referring to FIG. 5, the indexinformation 310 may include, for example, video indices 502, audioindices 504, text indices 506, parametric indices 508, and image contentindices 510. Referring now to FIG. 6, the index information 310 in themovie embodiment includes general index table(s) 602 that pertain to allmovies (such as actor biographies), and index table(s) 604, 606 specificto each movie. These tables 602, 604, 606 in FIG. 6 may each includeindices as shown in FIG. 5. The index information 310 specific to themovie embodiment is discussed in greater detail below.

Video Frame Pedigree

Referring now to the movie embodiment of the present invention, FIG. 7generally illustrates the making of a movie.

As is well known, during the production phase 406 a series of takes 710,712, 714 are shot on film 720 (typically 35 mm film in full-framesuper-35 format). Film 720 represents "negatives" and, thus, shall becalled "negative film 720" for reference purposes. A take is thesequence of frames (such as frame 704) from the clap of a clapboard(when the director says "action") to when the director says "cut" or"cut and print." There are many types of takes, such as a master take, aclose up (CU), an extreme close up (ECU), etc. Each take has one or moreshots, such as shot 706 in take 712. A shot represents those frames inthe take which are of value, and which may be used in the movie.

Each frame in the negative film 720 has an unique time code which can betied to the start of the respective take (i.e., from the clap of theclapboard). Thus, the frames in the negative film 720 can be identifiedby reference to time codes.

The negative film 720 is developed and portions are printed (they arenow positive) and are called "dailys." If the director ends a take bysaying "cut and print," then that take is printed. Otherwise, it is notprinted. FIG. 7 shows a portion of a daily 722. This daily 722 includesthe shot 706 containing the frame 704. Each frame in the dailys 722includes a unique time code. These time codes in the dailys 722 are thesame as the time codes in the negative film 720 as they were copied inthe printing process.

During the post-production phase 410, a theatrical presentation 724 ofthe movie is produced. The theatrical presentation 724 includes asequence of scenes, such as scene 708. Each scene includes one or moreshots, such as shot 706 in scene 708. The theatrical presentation 724 istypically generated by splicing together frames from the dailys, such asframe 704 from daily 722. This produces a work print of the whole movie.Using the workprint as a master, the negative film is cut and spliced tomatch. The cut negative is then printed to produce a timed (colorcorrected) positive. All the prints so far have been contact prints sothe original time codes are preserved. The theatrical negative (notshown) is produced optically (to change aspect ratio, make room forsound tracks, etc.) and new time codes are assigned for the whole movie.Theatrical presentation 724 prints are made from that negative. Eachframe in the theatrical presentation 724 includes an unique time codecopied from the theatrical negative. These time codes are measured fromthe beginning of the theatrical presentation 724. Thus, the time codesin the theatrical presentation 724 differ from the time codes in thedailys 722 and the negative film 720. However, the process forconverting from the time codes in the theatrical presentation 724 to thetime codes in the dailys 722 and negative film 720, and vice versa, iswell known.

Typically, versions of the movie other than the theatrical presentation724 are produced. One such version is the VCR video version 702. Videoruns at a different speed than film. Typically, video runs at 30 fps(frames per second) or 25 fps, while film runs at 24 fps. To compensatefor this difference, additional frames are added to the shots in the VCRvideo version 702. For example, a shot 706' in the VCR video version 702includes five frames, whereas the corresponding shot 706 in thetheatrical presentation 724 includes four frames. Each frame in the VCRvideo version 702 includes an unique time code. These time codes aremeasured from the beginning of the VCR video version 702. Since thenumber of frames per shot differ in the VCR video version 702 and thetheatrical presentation 724, the time codes between the VCR videoversion 702 and the theatrical presentation 724 also differ. However,the process for converting from the time codes in the VCR video version702 to time codes in the theatrical presentation 724, and to the timecodes in the dailys 722 and negative film 720, and vice versa, is wellknown.

Accordingly, it is possible to trace the lineage of any frame 704through the negative film 720, the dailys 722, the theatricalpresentation 724, the VCR video version 702, and any other version ofthe movie not shown in FIG. 7, by using the time code of the frame, andalso by using the well known time code conversion processes mentionedabove.

Source Table

FIG. 8 depicts a source table 802 according to a preferred embodiment ofthe present invention. A source table 802 exists for each version of amovie (i.e., the theatrical presentation, the VHS video version, theunrated version, the director's version, etc.), although the sourcetables 802 for different versions may be very similar to each other. Insome cases, different versions may have the same source table 802.

The source table 802 is similar to an edit decision list, but the sourcetable 802 contains information not found in a conventional edit decisionlist. Also, the present invention uses the source table 802 in ways thatdiffer from the ways that an edit decision list is used. Further,according to the present invention, the source table 802 is organizedand stored in electronic form. In fact, the source table 802 is part ofthe index information that is stored in the index information database122.

The source table 802 specifies the scenes, takes, and/or clips that arein a version of a movie. The source table 802 contains sufficientinformation so that the lineage of each frame in the movie can beidentified and located (an example of a frame's lineage or pedigree isshown in FIG. 7). The source table 802 represents a linkage mechanism sothat foundation information 108 related to each frame can be identified,located, and retrieved from the foundation information database 112.

The source table 802 includes a row for each scene, clip, and take inthe movie version. A clip is a sequence of frames taken without regardto scene, shot or take boundaries. However, it is taken from a singlesource with a uniform time base (such as theatrical presentation 724) sothat the start time code and end time codes have unique meanings. A typecode column 804 in the source table 802 identifies whether the row is ascene, clip, or take, and identifies the type of take (master, close up,extreme close up, etc.).

A first frame time code column 806 stores the time code of the firstframe of the item corresponding to the row. Thus, if the row correspondsto a scene, then the first frame time code column 806 stores the timecode of the first frame of the scene. The time code stored in the firstframe time code column 806 is measured from the beginning of the movieversion to which the source table 802 is associated. The presentinvention preferably uses SMPTE (Society of Motion Picture TelevisionExperts) time codes, but other time codes could alternatively be used.

A last frame time code column 808 stores the time code of the last frameof the item corresponding to the row. Thus, if the row corresponds to ascene, then the last frame time code column 808 stores the time code ofthe last frame of the scene. The time code stored in the last frame timecode column 808 is measured from the beginning of the movie version towhich the source table 802 is associated.

A frame count column 810 stores a number that is equal to the number offrames in the item corresponding to the row. Thus, if the rowcorresponds to a scene, and the scene has 79 frames, then the framecount column 810 stores "79."

A source title column 816 stores information identifying the source ofthe item corresponding to the row. Referring to FIG. 7, suppose that therow corresponds to the take 712. The source of this take 712 is thedaily 722 (actually, the ultimate source of take 712 is negative film720, but preferably the present invention utilizes positives wherepossible, rather than negatives and the time codes match). Thus,information identifying the daily 722 is stored in the source titlecolumn 816. As discussed above in conjunction with FIG. 3, the images onthe daily 722 were captured, digitized, and stored as foundationinformation 306 in the foundation information database 112. Preferably,what is stored in the source title column 816 is information sufficientto locate these digitized images of the daily 722 in the foundationinformation database 112 (such as a file name or address).

Note that the frames in a scene may originate from a number of sources.Thus, the source title column 816 is blank (or not applicable, N/A) forrows corresponding to scenes. Similarly, the source first frame timecode column 812 and the source frame count column 814 (described below)are also blank.

A source first frame time code column 812 stores the time code of thefirst frame of the item identified by the source title column 816.Assume, again, that the row corresponds to the take 712 in FIG. 7. Thesource first frame time code column 812 stores the time code of thefirst frame of the take 712. The time code stored in the source firstframe time code column 816 is measured from the beginning of the itemidentified by the source title column 816, i.e., from the beginning ofthe take 712.

A source frame count column 814 stores a number that is equal to thenumber of frames in the item identified by the source title column 816.

A "for which scene" column 818 contains information that identifies thescene in which the item corresponding to the row is used. For example,if the row corresponds to a take, then the "for which scene" column 818contains information that identifies the scene in which the take isused. This column is blank for rows corresponding to clips.

User-Driven Operation of the Present Invention

The user-driven operation of the present invention shall now bedescribed in conjunction with a flowchart depicted in FIG. 9A. This"user-driven" operation differs from the operation of the invention asshown in FIG. 3. Specifically, FIG. 3 illustrates the manner in whichthe present invention generates foundation information 306 and indexinformation 310. The user-driven operation depicted in FIG. 9A refers tothe manner in which the present invention access, retrieves, andpresents foundation information 306 from the foundation informationdatabase 112 to users in response to information requests and/or queriesfrom users.

For illustrative purposes, the user-driven operation of the presentinvention shall be described with reference to the movie embodiment. Itshould be understood, however, that the present invention is not limitedto the movie embodiment. The following description is applicable whenthe present invention is used in other applications.

The flowchart in FIG. 9A begins with step 950, where control immediatelypasses to step 952.

In step 952, in response to an appropriate request from an user via anuser device 106, the presentation and control component 104 transfers tothe user device 106 a list of items which the user may view and/orinteract with. Such items include, for example, various versions ofmovies, audio tracks related to movies, video games related to movies,text from books or magazines related to movies, etc. Such items alsoinclude personalized movie presentations and personalized moviecollections, which are described below.

The presentation and control component 104 preferably performs step 952by accessing the index information database 122 (via the index interfacecomponent 118) and/or the foundation information database 112 (via thefoundation information interface component 108) for a listing ofpossible viewing choices. This list of viewing choices is transferredfrom the presentation and control component 104 to the user device 106via the communication medium 124. The user device 106 displays this listof viewing choices in a well known manner.

In step 954, the presentation and control component 104 receives fromthe user (via the communication medium 124 and the user device 106)information identifying a particular item which the user wishes to view.

In step 956, the presentation and control component 104 accesses andretrieves the item identified by the user. In one embodiment, an ItemLocator table (which may be implemented as a collection of tables) isstored in the index information database 122. The Item Locator table hasa row for each of the possible viewing choices. Each row stores at least(1) the name of the item (or other information to unambiguously identifythe item), and (2) address information identifying the location in thefoundation information database 112 where the item is stored.

The presentation and control component 104 performs step 956 byaccessing the Item Locator table (via the index interface component118), locating the row corresponding to the item identified by the user,extracting the address information from this row, and using this addressinformation to access and retrieve the item from the foundationinformation database 112 (via the foundation information interfacecomponent 108). The item is transferred to the presentation and controlcomponent. 104 via the communication medium 124. For example, if theitem is a movie, then the digitized frames of the movie are transferredfrom the foundation information database 112 to the presentation andcontrol component 104.

In step 958, the presentation and control component 104 begins topresent the item to the user by transferring consecutive parts of theitem to the user device 106. It is noted that the item is transferred toa location specified by the user (such as the user's home). Also, theitem is preferably transferred to this location immediately after theuser has selected the item. Thus, the present invention implements truevideo on demand.

For illustrative purposes, the following discussion assumes that theitem is a movie. Thus, during step 958, the presentation and controlcomponent 104 sends the first digitized frame to the user device 106,where the frame is displayed.

In step 960, the presentation and control component 104 determineswhether it has received a query pertaining to the movie from the user(via the user device 106 and the communication medium 124). The mannerin which the user sends a query is described below. If a query has beenreceived, then the presentation and control component 104 processes thequery in step 962, and then proceeds to step 964. The manner in whichqueries are processed are described below. Step 964 is also processed ifno query was received.

In step 964, the presentation and control component 104 determines ifthere are any additional frames in the movie that have not yet been sentto the user. If additional frames exist, then control returns to step958 where the next frame is sent to the user device 106. Otherwise,operation of the flowchart of FIG. 9A is complete, as indicated by step966.

The above-described operation of the present invention can be modifiedin any number of ways to achieve better performance. For example,according to an alternate embodiment, the movie is sent directly to theuser device 106 from the foundation information database 112, ratherthan being sent first to the presentation and control component 104.Also, in this alternate embodiment, the entire movie is sent to the userdevice 106 all at one time, rather than being sent frame by frame.

In another alternate embodiment, personalized movie presentations andpersonalized movie collections are retrieved from the foundationinformation database 112 and presented to the user according to indexinformation contained in personalized edit decision lists. Personalizedmovie presentations and personalized movie collections are describedfurther below.

Sending and Interpreting User Queries

The user can send a query to the presentation and control component 104at any time while viewing and/or interacting with a movie (or some otheritem). Preferably, the user signals a query by pressing the "Pause"button provided by the user device 106 (this "Pause" button may besituated on the remote control unit, on the set top box, or on thecomputer keyboard, for example). By pressing the "Pause" button, theuser indicates that he has a question regarding the movie, scene, cut,and/or frame that is currently being displayed. The question may be, forexample, "Who is this person?," "What other movies has this directordone?," "Why is the street in this street wet when it appears to be aclear day?," or "What other movies have script lines similar to what wasjust said?" The questions which users are allowed to ask areimplementation dependent. Thus, the particular questions discussedherein are presented for illustrative purposes only, and are notlimiting.

Preferably, the user identifies his specific question by traversing anuser menu hierarchy. User menus are well known. Thus, the specifics ofthe user menu hierarchy used to identify user questions shall not bediscussed in detail.

It is noted, however, that the user menu may be driven and controlled bythe presentation and control component 104 by exchanging control anddata signals between the presentation and control component 104 and theuser device 106. Alternatively, software control logic and/or hardwarecontrol logic can be incorporated into the user device 106 such that theuser device 106 drives and controls the user menu.

Note that the question "Who is this person?" is somewhat ambiguous, inthat it could mean "Who is the principle actor?," or "Who is the stuntdouble actor?," or "Who is the character?" In one embodiment of theinvention, this ambiguity is solved by having a very detailed menustructure, so that the precise question is identified by the user.However, users may find this approach somewhat tedious.

Accordingly, to an alternate embodiment, the present invention makesassumptions as to what the users' precise questions are. For example, ifthe user asks "Who is this person?," the present invention may assumethat the user is really asking "Who is the character in this scene?" Theassumptions made by the present invention in this alternate embodimentare implementation dependent and are based on many factors, such as thecontext in which the question was asked, market information, surveys,empirical use, etc. For illustrative purposes, this alternate embodimentof the invention (where assumptions are made as to the user's precisequestion) is described below. The assumptions that are employed in thefollowing description are used only by way of example, and may differdepending on the particular implementation.

Processing User Queries

FIG. 9B presents a flowchart depicting the manner in which the presentinvention processes an user query (i.e., the manner in which the presentinvention performs step 962 in FIG. 9A). The flowchart in FIG. 9B beginswith step 902, where control immediately passes to step 904.

In step 904, the user device 106 determines the time code correspondingto the current frame (this time code is called the "current frame timecode" for reference purposes). The user device 106 then transfers thecurrent frame time code to the presentation and control component 104via the communication medium 124. (The user device 106 may send thecurrent frame time code such that the presentation and control component104 receives it at the same time as receiving the user query, i.e., atstep 960 in FIG. 9A.)

The "current frame" represents the frame that was being presented on theuser device when the user issued the query (i.e., pressed the "Pause"button). As noted above, clock information 420 (i.e., time codeinformation) is embedded in the video data 416 of the movie (see FIG.4). Accordingly, in step 904, the user device 106 extracts the currentframe time code corresponding to the current frame from the video data416 of the movie being presented to the user.

The current frame time code is measured from the start of the movie(rather than being measured from a daily 722 or negative film 720, forexample). Thus, time code conversions are performed in subsequent steps.

In step 906, the presentation and control component 104 accesses theindex information database 122 (via the index interface component 118)and extracts from the source table 802 associated with the movie beingpresented (called the current movie for reference purposes) all rowswhere the current frame time code is greater than or equal to the firstframe time code, and the current frame time code is less than or equalto the last frame time code.

In step 908, the presentation and control component 104 identifies thequery type or category. This query type is indicated by informationcontained in the user query received in step 960 of FIG. 9A (it isassumed that the user selected a query type by traversing an user menuhierarchy, as discussed above). Each implementation of the presentinvention supports a set of query types. For illustrative purposes, aselected set of query types is discussed below. Other query types willbe apparent to persons skilled in the relevant art(s).

In step 910, the presentation and control component 104 processes theuser query according to the type of the query as identified in step 908.The manner in which the presentation and control component 104 processesa selected set of query types is discussed below. The processing ofother query types will be apparent to persons skilled in the relevantart(s).

The flowchart of FIG. 9B is complete after step 910 is performed, asindicated by step 912.

Query Type: Who Is This?

This section discusses the "who is this?" query type. According to anembodiment of the present invention, the index information database 122stores at least three index tables to support the processing of thisquery type: an Actor In Take table 1002, a Character In Scene Table1012, and an Actor Play Character table 1018. These tables 1002, 1012,1018 represent movie specific tables, and are shown in FIGS. 10A-10C.

The Actor In Take table 1002 includes at least one row for each actorappearing in each take. An actor column 1004 stores the name of theactor, a take title column 1006 stores the name of the take, a starttime code column 1008 stores the time code of the first frame of thetake in the current movie, and the end time code column 1010 stores thetime code of the last frame of the take in the current movie. The timecodes in the start time code column 1008 and the end time code column1010 are measured from the start of the current movie. Information forthe Actor In Take table 1002 is derived by the index generator component308 by reference to production schedule information (that is part of theproduction information 408).

The Character In Scene table 1012 includes a row for each character whoappears in each scene. A scene column 1014 stores the name of the scene,and a character column 1016 stores the name of a character who appearsin the scene. Information for the Character In Scene table 1012 isderived by the index generator component 308 by reference to the script.

The Actor Plays Character table 1018 includes, for each character in thecurrent movie, a row for each actor who plays the character. A charactercolumn 1020 stores the name of the character, an actor column 1022stores the name of an actor who plays the character, and a code column1024 identifies the manner in which the actor played the character(i.e., as the principle actor, as the stunt double, as the body double,etc.). Information for the Actor Plays Character table 1018 is derivedby the index generator component 308 by reference to the credits of themovie and the production schedule.

FIG. 11 depicts a flowchart representing the manner in which the presentinvention processes a "who is it?" query according to one embodiment.This flowchart begins with step 1102, where control immediately passesto step 1106.

In step 1106, the presentation and control component 104 retrieves fromthe Actor In Take table 1002 rows that relate to the current frame. Step1106 is performed by reference to the rows that were retrieved from thesource table 802 in step 906. Step 1106 is performed by retrieving fromthe Actor In Take table 1002 all rows having a start time code (fromcolumn 1008) equal to the start time code (from column 806 in the sourcetable 802) in the take row retrieved in step 906, and also having an endtime code (from column 1010) equal to the end time code (from column 808in the source table 802) in the take row retrieved in step 906. Theserows from the Actor In Take table 1002 identify the actors who appear inthe current frame.

Step 1104 tests to see that at least one actor 1004 was included in thattake. (It might have been a long scenery shot for example.) If no actorsare found in the take, then the scope is widened to include all thecharacters in the entire scene and processing continues at step 1120.Otherwise, processing continues with step 1108.

In step 1108, the presentation and control component 104 transfers tothe user device 106 via the communication medium 124 the names of theactors stored in the actor column 1004 of these rows retrieved from theActor In Take table 1002. The user device 106 presents these names tothe user in a well known manner.

In step 1110, the presentation and control component 104 retrieves allrows from the Actor Plays Character table 1018 associated with allactors identified in step 1106 (i.e., actors corresponding to rowsretrieved from the Actor In Take Table 1002). Step 1110 is performed byreference to the actor column 1022 in the Actor Plays Character table1018.

In step 1112, the presentation and control component 104 transfers tothe user device 106 via the communication medium 124 the names of thecharacters stored in the character column 1020 of these rows retrievedfrom the Actor Plays Character table 1018. The user device 106 presentsthese names to the user in a well known manner.

In step 1114, the presentation and control component 104 retrieves fromthe Actor Plays Character table 1018 all rows with code 1024 value of"principal" related to the characters identified in step 1110 (i.e., thecharacters corresponding to the rows retrieved from the Actor PlaysCharacter table 1018). Step 1114 is performed by retrieving from theActor Plays Character table 1018 all rows corresponding to theseidentified characters.

In step 1116, the presentation and control component 104 transfers tothe user device 106 via the communication medium 124 the names of theactors (column 1022) and their roles (column 1024) from these rowsretrieved from the Actor Plays Character table 1018. The user device 106presents this information to the user in a well known manner.

If, in step 1104, it is determined that no actor may be in the currentframe, then step 1120 is performed. In step 1120, the presentation andcontrol component 104 retrieves all rows from the Character In Scenetable 1012 associated with characters appearing in the current scene.The "current scene" is the scene in which the current frame iscontained, and is identified by reference to information stored in the"for which scene" column 818 in the take row previously retrieved fromthe source table 802 (in step 906). Step 1120 is performed by retrievingfrom the Character In Scene table 1012 all rows corresponding to thecurrent scene (by reference to the scene column 1014 in the Character InScene table 1012).

In step 1122, the presentation and control component 104 transfers tothe user device 106 via the communication medium 124 the names of thecharacters stored in the character column 1016 of these rows retrievedfrom the Character In Scene table 1012. The user device 106 presentsthese names to the user in a well known manner. After step 1122 isperformed, then control moves to step 1114, described above (although,in this case, step 1114 is performed based on the results of step 1120).

The operation of the flowchart in FIG. 11 is complete after step 1116 isperformed, as indicated by step 1118.

Query Type: What Other Films Has This Director Worked On?

This section discusses the "what other films has this director workedon?" query type. According to an embodiment of the present invention,the index information database 122 stores at least two index tables tosupport the processing of this query type: an One Movie Credit table1202 (which is movie specific) and an All Movies Credit table 1208(which applies to all movies). These tables 1202, 1208 are shown inFIGS. 12A and 12B.

The One Movie Credit table 1202 includes at least one row for eachperson involved in the creation of the movie (for example, a row foreach person listed in the credits). Each row includes a person column1204 storing the name of the person, and a job column 1206 indicatingthe person's position. If a person had many jobs, then there is one rowin the One Movie Credit table 1202 for each job. Information for the OneMovie Credit table 1202 is derived by the index generator component 308by reference to the credits of the movie, and also by reference tocontracts (pre-production information 404).

The All Movies Credit table 1208 includes one row for each job that eachperson had in each movie represented in the foundation informationdatabase 112. A movie title column 1212 stores a movie's name, a personcolumn 1214 stores a person's name, a job column 1216 stores a job thatthe person had in the movie, and a table identifier row 1210 storesinformation that identifies where other index tables specific to themovie are stored in the index information database 122. Information forthe All Movies Credit table 1208 is derived by the index generatorcomponent 308 by reference to the credits of the movie, and also byreference to contracts (pre-production information 404).

The present invention processes a "what other films has this directorworked on?" query type as follows. First, the presentation and controlcomponent 104 accesses the One Movie Credit table 1202 for the currentmovie and retrieves the row where the job column 1206 is equal to"Director." Second, the presentation and control component 104 extractsthe name of the director from the person column 1204 of this row. Third,the presentation and control component 104 accesses the All MoviesCredit table 1208 and retrieves all rows where the person column 1214 isequal to the director's name. Fourth, the presentation and controlcomponent 104 extracts the movie title (from the movie title column1212) and the job (from the job column 1216) from these rows and sendsthem to the user device 106 via the communication medium 124. The userdevice 106 displays this information in a well known manner.

Query Type: When Did This Happen?

This section discusses the "when did this happen?" query type. Accordingto an embodiment of the present invention, the index informationdatabase 122 stores at least two index tables to support the processingof this query type: a Time In Story table 1302 and a Take Details table1312, both of which are movie specific. These tables 1302, 1312 areshown in FIGS. 13A and 13B.

The Time In Story Table 1302 has a row for each scene in the currentmovie. A scene column 1304 stores the name of the scene, a time sincestory start column 1306 that stores a value representing the time (interms of the story) that has elapsed since the current movie started, atime since last scene column 1308 that stores a value representing thetime (in terms of the story) since the last scene ended, and an absolutedate/time column 1310 that stores a value that represents the date andtime (in terms of the story) of the events depicted in the currentscene. Information for the Time In Story table 1302 is derived by theindex generator component 308 by reference to time references in thescript (for example, time indicators such as "that night," "later thatday," "next morning," etc.).

The take details table 1312 has a row for each take. A take title column1314 stores the title of the take, a take first frame time code column1316 stores the time code of the take measured from the beginning of thecurrent movie, a location name column 1318 stores the name of thelocation where the take was shot, a date shot (created) column 1320 anda time shot (created) column 1322 store a date and time when the takewas shot, and a method column 1324 stores information identifying themanner in which the take was shot (i.e., steady-cam, digital morph,etc.). Information for the Take Details Table 1312 is derived by theindex generator component 308 by reference to the production schedule.

The present invention processes a "when did this happen?" query type asfollows. First, the presentation and control component 104 identifiesthe current scene by reference to the "for which scene" column 818 inthe take row previously retrieved from the source table 802. Second, thepresentation and control component 104 retrieves from the Time In StoryTable 1302 the row where the scene column 1304 is equal to the name ofthe current scene. Third, the presentation and control component 104extracts the time and date information from columns 1306, 1308, and 1310of this row, and forwards this information to the user device 106 viathe communication medium 124. The user device 106 displays thisinformation in a well known manner.

The present invention alternatively processes a "when did this happen?"query type as follows. First, the presentation and control component 104retrieves from the Take Details table 1312 the row where the value inthe take first frame time code column 1316 is equal to the value in thefirst frame time code 806 of the take row previously retrieved from thesource table 802. Second, the presentation and control component 104extracts the date and time information from columns 1320 and 1322 ofthis row, and forwards this information to the user device 106 via thecommunication medium 124. The user device 106 displays this informationin a well known manner.

Other queries can be answered by reference to the Take Details Table1312. For example, a "Where was this take shot?" query can be answeredby reference to the location name column 1318, and a "How was this takeshot?" query can be answered by reference to the method column 1324.

Query Type: Why Queries

A "why" query is supported only if a corresponding answer is available.An example why question is: "Why is the street in this scene wet when itappears to be a clear day?" A typical answer is: "The director ofphotography uses water to create more interesting images on film bycapturing the reflections of other parts of the scene in the street. Thedirector calls for a wet-down just before the shot and a water truckdrives through the scene and makes it wet."

The tables and methods for answering these "why" questions are shown inFIG. 14A, FIG. 14B and FIG. 15. Preferably, the present inventionincludes a Question/Answer table 1404 and Question Applies to Scenetable 1424 to support "why" questions. These tables 1404, 1424 are moviespecific, and are stored in the index information database 122.Information for the Question/Answer table 1404 and Question Applies toScene table 1424 is derived by the index generator component 308 byreference to many sources, such as interviews with persons involved inthe making of the movie, diaries kept during the making of the movie,books describing the making of the movie, etc. Preferably, a humanoperator uses this information to compose "why" questions and theiranswers, and to insert these questions and answers in theQuestion/Answer table 1404. Similarly, a human operator uses thisinformation to determine appropriateness of each question to particularscenes or sequences of scenes and inserts this information into theQuestion Applies to Scene table 1424.

The Question/Answer table 1404 includes a row for each why question thatis supported. A Question Number column 1406 uniquely identifies thequestion, a question column 1408 stores the why question, and an answercolumn 1410 stores an answer to the question.

The Question Applies to Scene table 1424 includes a row for each segmentof a movie to which a question applies. A Question Number column 1426identifies the question, a First Scene column 1428 identifies the firstscene in a sequence of scenes for which this question is appropriate,and a Last Scene column 1430 identifies the last scene in a sequence ofscenes for which this scene is appropriate. Since questions can apply toseveral scenes in a movie, many rows may have the same Question Number1426. Many questions will apply to any given scene so First Scene 1428and Last Scene 1430 values may also be repeated.

The present invention processes a why question as is shown in FIG. 15.The procedure gets control at 1510 and proceeds immediately to the firstprocessing step 1520. First the presentation and control component 104accesses the row of source table 802 where the type code 804 is "scene"and the current time code of the movie is between the First Frame TimeCode 806 and the Last Frame Time Code 808 to retrieve the correct forwhich scene 818 value. Second, step 1522, the Question Applies to Scenetable is accessed to retrieve a list of question numbers 1426 where thepreviously retrieved for which scene 818 value is between the FirstScene 1428 and Last Scene 1430. There is a possibility of duplicatequestion number 1426 values in the list and these should be removed.Third, step 1512, the questions 1408 from the Question/Answer table 1404along with the questions numbers 1406 are forwarded to the user device106 through the communication medium 124 and are subsequently displayedin a well known manner. Upon receipt of a response from the user device106, the question 1408 and answer 1410 corresponding to the questionnumber 1406 selected by the user are retrieved 1514. Finally, step 1516,the question 1408 and answer 1410 are forwarded to the user device 106via the communication medium 124. The user device 106 displays thisinformation in a well known manner. Processing then proceeds directlythrough termination 1518.

Show Option

According to an embodiment of the invention, the user can request thatthe script be displayed as the movie is presented. The present inventionprocesses this user request as follows. First, the time code for thecurrent frame is determined. Second, the row corresponding to thecurrent scene is retrieved from the source table 802. This is done byidentifying a scene row in the source table 802 where the current frametime code is greater than or equal to the value of the first frame timecode column 806, and less than or equal to the value of the last frametime code column 808. Based on the information contained in this scenerow, it is possible to synchronize the display of the movie with thedisplay of the script (the script is retrieved from the foundationinformation database 112).

The operation of the show option shall now be discussed in greaterdetail. The identified row (described above) is the first of asequential set of scene rows which describe the rest of the movie. Eachhas values for frame count 810 and for which scene 818. The Script textis accessed using the for which scene 818 values obtained from thesource table 802 as index keys. The frame count 810 from each row tellsexactly how long that scene will be on the screen. The scrolling rate ofthe script text is adjusted so that the script rolls by in the sameamount of time that the scene takes in the movie. Each new segment ofthe script is accessed to have it ready when it has to start appearingon the screen. Since the scrolling window has many lines of the scripton it, the slug line for the following scene will appear before thevideo for the current scene has completed. Well known techniques foranticipatory buffering enable smooth scrolling and synchronization.

Manager

As noted above, the present invention includes a manager 1602 which isshown in block diagram form in FIG. 16. The manager 1602 includes one ormore capture and digitizer components 304 (for simplicity purposes, onlyone is described), one or more index generation components 308 (forsimplicity purposes, only one is described), a personalized presentationmanager 1604, and a personalized collection manager 1606. The captureand digitizer component 304 and the index generation component 308 weredescribed above. The personalized presentation manager 1604 and thepersonalized collection manager 1606 shall now be discussed.

Personalized Presentation Manager

The personalized presentation manager 1604 enables a user to create apersonalized version of a movie. The user, in his personal movieversion, may include shots not in the theatrical presentation or the VCRvideo version, and/or may exclude shots, takes, clips, and/or scenesthat are in the theatrical presentation and/or the VCR video version.The user may change the sequence of shots, takes, clips, and/or scenes,and/or may change the length of shots, takes, clips, and/or scenes.Other modification options available to users will be apparent topersons skilled in the relevant art based on the discussion containedherein.

The user interacts with the personalized presentation manager 1604 tocreate a personalized movie version. In one embodiment, the personalizedpresentation manager 1604 forms part of the presentation and controlcomponent 104, such that interaction with the user is achieved throughthe transfer of control and data messages between the presentation andcontrol component 104 and the user device 106. In an alternateembodiment, the personalized presentation manager 1604 is part of thepresentation and control component 104, but becomes resident (at leastin part) at the user device 106 pursuant to a user request to create oredit a personalized movie version. This alternate embodiment isadvantageous in that it reduces traffic on the communication medium 124.Other ways for implementing the personalized presentation manager 1604will be apparent to persons skilled in the relevant art(s).

FIG. 17A presents a flowchart depicting the operation of thepersonalized presentation manager 1604 when working with an user tocreate or modify a personalized movie version. This flowchart beginswith step 1702, where control immediately passes to step 1704.

In step 1704, the personalized presentation manager 1604 creates a copyof the edit decision list corresponding to the movie which the userwishes to personalize. This copy is called the personalized editdecision list for reference purposes. An example of a personalized editdecision list 1702 is shown in FIG. 1702.

According to a preferred embodiment of the present invention, apersonalized movie version is made up of a sequence of clips.Accordingly, the personalized edit decision list 1702 includes a row foreach clip. The columns in the personalized edit decision list 1702 havethe same function as those of the same name in the source table 802(FIG. 8). Thus, once editing of the personalized edit decision list 1702has been completed, and the personalized movie is being viewed, it ispossible to process user queries in the same manner as discussed above.

It may be necessary to perform a translation from the movie's editdecision list, which includes scene and take entries, to thepersonalized edit decision list 1702, which contains just clip entries.The mechanics of this translation will be apparent to persons skilled inthe relevant art(s).

In step 1706, the personalized presentation manager 1604 enables theuser to modify the personalized edit decision list 1702. In thiscapacity, the personalized presentation manager 1604 acts as an editor,such that the user can edit the rows in the personalized edit decisionlist 1702. However, the personalized presentation manager 1604 functionsas more than an editor, since it allows a user to view any shot, take,scene, clip, etc., contained in any version of the movie, or in theunderlying dailys 722 or even the underlying negative film 720. Theseshots, takes, scenes, clips, etc., are located by using the associatedtime codes as a reference to the source table 802 for the movie. Basedon information contained in the source table 802, it is possible toidentify the storage location in the foundation information database 112of any shot, take, scene, clip, etc., as discussed above. Accordingly,the user can view his proposed change to the personalized edit decisionlist 1702 before actually making the change.

The operation of the flowchart in FIG. 17A is complete after step 1706is performed, as indicated by step 1708.

FIG. 18 presents a flowchart representing the manner in which thepresentation and control component 104 presents a personalized movie toa user. This flowchart begins with step 1802, where control immediatelypasses to step 1804.

In step 1804, the presentation and control component 104 selects andretrieves the first entry (row) from the personalized edit decision list1702 for the personalized movie the user wants to view.

In step 1806, the presentation and control component 104 uses theinformation contained in this row to retrieve from the foundationinformation database 112 the digitized frames corresponding to the cliprepresented by this row (that was retrieved in step 1804). Inparticular, the presentation and control component 104 uses informationfrom the source title column 1712, the source first frame time codecolumn 1708, and the source frame count column 1710 from this row(retrieved from the personalized edit table in step 1804) to obtainthese digitized frames.

In step 1808, the presentation and control component 104 presents thesedigitized frames to the user. In particular, the presentation andcontrol component 104 transfers these digitized frames to the userdevice 106 via the communication medium 124, and the user device 106displays the frames in a well known manner. Note that, while theseframes are being presented to the user, the user can issue a query, asdescribed above. Any such query issued by the user is processed in thesame manner as described above.

In step 1810, the presentation and control component 104 determineswhether there are any more entries in the personalized edit decisionlist 1702 left to process. If there are more entries to process, then instep 1814 the next entry is selected, and control returns to step 1806.Otherwise, operation of the flowchart in FIG. 18 is complete asindicated by step 1812.

Personalized Collection Manager

The personalized collection manager 1606 enables a user to create apersonalized collection of items for later presentation. These items maybe scenes and/or clips from a movie or from multiple movies, tracks froma soundtrack or multiple soundtracks, favorite "coming attraction"clips, etc., or any combination of these. Thus, the personalizedcollection manager 1606 enables a user to create collections orgroupings of items of interest.

The user interacts with the personalized collection manager 1606 tocreate a personalized collection. In one embodiment, the personalizedcollection manager 1606 forms part of the presentation and controlcomponent 104, such that interaction with the user is achieved throughthe transfer of control and data messages between the presentation andcontrol component 104 and the user device 106. In an alternateembodiment, the personalized collection manager 1606 is part of thepresentation and control component 104, but becomes resident (at leastin part) at the user device 106 pursuant to a user request to create oredit a personalized collection. This alternate embodiment isadvantageous in that it reduces traffic on the communication medium 124.Other ways for implementing the personalized collection manager 1606will be apparent to persons skilled in the relevant art(s).

FIG. 19 presents a flowchart depicting the operation of the personalizedcollection manager 1606 when working with an user to create or modify apersonalized collection. This flowchart begins with step 1902, wherecontrol immediately passes to step 1904.

In step 1904, the personalized collection manager 1606 creates a newpersonalized collection table, or opens an existing personalizedcollection table for editing (which action is taken depends on theuser's request). FIG. 20 illustrates an example personalized collectiontable 2002, which has an entry (row) for each item in the collection.The source first frame time code column 2004, source last frame timecode column 2006, and source title column 2008 are similar to theidentically named columns in the source table 802 (FIG. 8). Thesecolumns 2004, 2006, 2008 contain information necessary to locate theunderlying foundation information in the foundation information database112. The personalized collection table 2002 shown in FIG. 20 is somewhatspecialized for takes, cuts, scenes, and/or clips from a movie. Minormodifications to the personalized collection table 2002 may be necessaryto accommodate other items, such as audio tracks (for example, for audiotracks, the source first frame time code column 2004 and source lastframe time code column 2006 are modified to store time codes or otherindicia to identify the beginning and end of an audio track). The natureand implementation of such modifications will be apparent to personsskilled in the relevant art(s).

In step 1906, the personalized collection manager 1606 enables the userto modify the personalized collection table 2002. In this capacity, thepersonalized collection manager 606 acts as an editor, such that theuser can edit the rows in the personalized collection table 2002.However, the personalized collection manager 606 functions as more thanan editor, since it allows a user to preview any shot, take, scene,clip, audio segment, etc., contained in any version of the movie, theunderlying dailys 722, the underlying negative film 720, or otherfoundation information stored in the foundation information database112. These shots, takes, scenes, clips, audio segments, etc., arelocated by using the associated time codes or other identifying indiciaas a reference to the source table 802 for the movie, or to other indextables stored in the index information database 122. Based on this indexinformation, it is possible to identify the storage location in thefoundation information database 112 of any shot, take, scene, clip,audio segment, etc., as discussed above. Accordingly, the user can viewhis proposed change to the personalized collection table 2002 beforeactually making the change.

The operation of the flowchart in FIG. 19 is complete after step 1906 isperformed, as indicated by step 1908.

The presentation and control component 104 presents a personalizedcollection by traversing each entry in the corresponding personalizedcollection table 2002 in a manner similar to that discussed above inconjunction with FIG. 18. While viewing a personalized collection, anuser can issue a query as discussed above. The present inventionprocesses such a query in the manner discussed above.

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample only, and not limitation. Thus, the breadth and scope of thepresent invention should not be limited by any of the above-describedexemplary embodiments, but should be defined only in accordance with thefollowing claims and their equivalents.

While the invention has been particularly shown and described withreference to (a) preferred embodiment(s) thereof, it will be understoodby those skilled in the art that (various changes) (the foregoing andother changes) in form and details may be made therein without departingfrom the spirit and scope of the invention.

Having thus described our invention, what we claim as new and desire tosecure by Letters Patent is:
 1. A computer program product comprising acomputer readable medium having computer program logic recorded thereonfor enabling a processor in a computer system to allow a user to view apersonalized version of a movie, and to allow the user to query afoundation information database while the personalized version of themovie is being presented to the user, the foundation informationdatabase storing primary information and secondary informationpertaining to content and creation of the movie, said computer programproduct comprising:movie presenting means, responsive to a request fromthe user, for enabling said processor to present to the user thepersonalized version of the movie; means for enabling said processor toallow the user to query the foundation information database forinformation pertaining to the content and creation of the movie whilethe personalized version of the movie is being presented to the user;frame identification means for enabling said processor to identify aframe of the personalized version of the movie that was being presentedto the user when the user issued said query; secondary informationidentification means for enabling said processor to identify portions ofthe secondary information in the foundation information databasepertaining to the content and creation of the personalized version ofthe movie as identified by said frame and as specified by said query;secondary information retrieval means for enabling said processor toretrieve from the foundation information database said portions of thesecondary information; and means for enabling said processor to presentto the user said retrieved portions of the secondary information.
 2. Thecomputer program product of claim 1, wherein the primary informationincludes post-production information representing the movie itself, andthe secondary information includes pre-production and productioninformation pertaining to content and creation of the movie.
 3. Thecomputer program product of claim 2, wherein said movie presenting meansretrieves the primary information from the foundation informationdatabase when presenting the personalized version of the movie to theuser.
 4. The computer program product of claim 1, wherein said moviepresenting means presents to the user the personalized version of themovie in accordance with a modified edit decision list that defines saidpersonalized version of the movie, said modified edit decision listincluding an entry for each clip in said personalized version of saidmovie, each entry storing first information identifying time codes offrames in a clip, and second information needed to locate primary andsecondary information pertaining to creation and content of said clip.5. The computer program product of claim 4, wherein said frameidentification means comprises:means for enabling said processor toextract from said personalized version of said movie a time code of saidframe that was being presented to said user when said user issued saidquery.
 6. The computer program product of claim 5, wherein saidsecondary information identification means comprises:means for enablingsaid processor to compare said time code of said frame with said firstinformation stored in entries of said modified edit decision list toidentify at least one entry of said modified edit decision list relatingto said frame; and information utilization means for enabling saidprocessor to utilize said first and second information stored in said atleast one entry to locate secondary information related to said frame.7. The computer program product of claim 5, in which said foundationinformation database comprises:an actor in take table having an entryfor each actor appearing in each take of said movie, each entry in saidactor in take table storing information identifying an actor,information identifying a take, and information identifying time codesof frames in said movie corresponding to said identified take; acharacter in scene table having an entry for each character appearing ineach scene of said movie, each entry in said character in scene tablestoring information identifying a scene, and information identifying acharacter who appears in said identified scene; and an actor playscharacter table having an entry for each actor who portrays eachcharacter in said movie, each entry in said actor plays character tablestoring information identifying a character in said movie, informationidentifying an actor who plays said character in said movie, andinformation identifying a manner in which said identified actor playedsaid identified character in said movie;wherein said informationutilization means comprises: means for enabling said processor toutilize said first information stored in said at least one entry toidentify and retrieve from said actor in take table entries identifyingactors appearing in a take encompassing said frame.
 8. The computerprogram product of claim 7, wherein said information utilization meansfurther comprises:means for enabling said processor to identify andretrieve from said actor plays character table entries identifyingcharacters played by said identified actors.
 9. The computer programproduct of claim 8, wherein said information utilization means furthercomprises:means for enabling said processor to identify and retrievefrom said character in scene table entries identifying scenes in whichsaid identified characters appear.
 10. The computer program product ofclaim 1, further comprising:means for enabling said processor to allowthe user to modify a copy of an edit decision list associated with themovie to thereby generate said modified edit decision list.
 11. Thecomputer program product of claim 1, further comprising:means forenabling said processor to charge the user for each query issued by theuser and processed by said processor.
 12. A computer program productcomprising a computer readable medium having computer program logicrecorded thereon for enabling a processor in a computer system to allowa user to view a personalized collection of audio/visual items, and toallow the user to query a foundation information database while thepersonalized collection of audio/visual items is being presented to theuser, the foundation information database storing primary informationand secondary information pertaining to content and creation of thecollection, said computer program product comprising:collectionpresenting means, responsive to a request from the user, for enablingsaid processor to present to the user the personalized collection ofaudio/visual items; means for enabling said processor to allow the userto query the foundation information database for information pertainingto the content and creation of the collection while the personalizedcollection of audio/visual items is being presented to the user; itemidentification means for enabling said processor to identify an item ofthe personalized collection of audio/visual items that was beingpresented to the user when the user issued said query; secondaryinformation identification means for enabling said processor to identifyportions of the secondary information in the foundation informationdatabase pertaining to the content and creation of the personalizedcollection of audio/visual items as identified by said item and asspecified by said query; secondary information retrieval means forenabling said processor to retrieve from the foundation informationdatabase said portions of the secondary information; and means forenabling said processor to present to the user said retrieved portionsof the secondary information.
 13. The computer program product of claim12, wherein the personalized collection of audio/visual items is apersonalized version of a movie.
 14. The computer program product ofclaim 12, wherein said collection presenting means presents to the userthe collection in accordance with a modified edit decision list thatdefines said collection, said modified edit decision list including anentry for each item in said collection, each entry storing informationneeded to locate primary and secondary information pertaining tocreation and content of an item.
 15. The computer program product ofclaim 14, further comprising:means for enabling said processor to allowthe user to modify a copy of an edit decision list associated with thecollection to thereby generate said modified edit decision list.
 16. Thecomputer program product of claim 12, further comprising:means forenabling said processor to charge the user for each query issued by theuser and processed by said processor.
 17. The computer program productof claim 12, wherein said collection presenting means retrieves theprimary information from the foundation information database whenpresenting the personalized collection to the user.
 18. The computerprogram product of claim 12, wherein said personalized collectionincludes any combination of shots, takes, scenes, clips, frames, andaudio segments.
 19. The computer program product of claim 12, whereinsaid collection presenting means comprises:means for sequentiallyretrieving primary information from the foundation information databasecorresponding to each item of the personalized collection, and forpresenting the retrieved primary information to the user.